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ABSTRACT 

The problem is defined as examining a list of the 
research projects presently sponsored by the Department 
of the Navy under GOR-43 and analyze the ordered project 
listings as ranked by panels composed of persons with 

interest in those projects. 

Data consisted of project priority rankings of 193 
judges composing 9 separate panels, each judge ranking 
the projects twice in priority order; once considering 
the practical significance of the project and once con- 
sidering the possibility of a successful research effort 
Project rankings of judges within each panel were scaled 
using the FORD procedure and the 9 panel rankings then 
investigated for similarities using methods of cluster 

analy s is . 

The results showed that no 2 judge panels submitted 
highly correlated rankings of the 18 projects, but that 
under each criterion several project clusters were evi- 
dent, indicating equal research emphasis desired by the 

panels . 
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INTRODUCTION 



I . 

There exists within the Department of the Navy, many 
organizations with varying convictions about which Navy 
Personnel Research and Navy Training Research Programs 
should receive priority when allocating research funds 
and research time. The diverse opinions of program empha- 
sis are due partly to the different compositions of the 
groups, partly to the different functions of the groups 
and partly to the relationships of the groups to other 
organizations both inside and outside the Department of 
the Navy. 

Undoubtedly, those who make the final decisions have 
the greater number of relevant facts upon which to base 
their decision. They probably exert greater effort to 
the decision-making process itself than their critics 
might. Thus their decisions should be the best "all fac- 
tors considered", decisions. Nevertheless, an examination 
of the diversity of opinions among different groups 
should help their decision-making process by providing 
additional information and by pointing out areas where 
closer agreement among various groups would promote the 
research effort. 

The objective of this paper is to draw, from various 
groups, data concerning which programs they feel should 
be studied, analyze the group decisions and examine 
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similarities and. dissimilarities of their priority pro- 
ject choices • 

The nature of the problem is two-fold. 

1. Compare the groups by comparing their choices of 
which projects should receive priority when allocating 
research laboratory resources. Establish communities of 
judge groups whose priority rankings displayed a degree 
of similarity. 

2. Examine a list of projects currently being 
studied by the Bureau of Naval Personnel and associated 
research laboratories. Determine if there exists projects 
which inter-group consensus shows should receive either 
high or low priority when allocating research laboratory 
resources . 

The method chosen to compare the judge groups was by 
means of a two-dimensional plot where inter -point dis- 
tances on the plot are a monotonic function of the judge 
group similarities. More specifically, 

r . . > r . , - > d . . < d . , 

ij jk ij jk 

r__ = a correlation measure between groups i and j. 

d_ = the inter-point distance for points i and j on the 
two-dimensional plot. 

By examining the resulting plots using methods of 
cluster analysis, those judge groups whose project prior- 
ity rankings compared favorably were readily evident. 
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The methods for comparing the various projects were 
the same as for the judge groups except that a dissimi- 
larity vice correlation measure was utilized for inter- 
object comparison. The technique for determining the 
plot accepted either statistic as input. The resulting 
plot displayed the projects as an inverse monotonic func- 
tion of the dissimilarity measure. Specifically, 



t . . < t => d . . < d .. 
ij jk ij jk 

t^ = the measure of dissimilarity between projects i and 
j, d_ = the inter-point distance for points i and j on 
the two-dimensional plot. 

Methods of cluster analysis discerned those projects 
which received similar rankings on the priority scales 
of the judge groups. 

Comparison of the mean ranking for each project over 
all judge groups showed which projects were ranked high, 
low, etc. The variance of the mean ranking for each pro- 
ject indicated the consistency of its ranking by the 
judge groups, e.g., the projects with the lowest rank 
variance were ranked most consistently by all groups. 
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II. DATA DESCRIPTION 



The data source for this study consisted of approxi- 
mately two-hundred individual judges, each ranking the 
same eighteen projects on a priority ladder. That pro- 
ject which the judge felt should receive top priority was 
ranked first; the second most important was ranked second, 
etc. In addition, each judge was allowed to indicate 
programs which he believed should receive equal emphasis 
by placing them (up to six) in the same group. To in- 
crease the validity of the data, judges were asked to 
rank only the projects which they felt knowledgeable 
about and competent to rank. 

Each judge produced two priority ladders, one ranked 
using the criterion of Military Significance of the pro- 
ject and one ranked using the criterion of Investment 
Risk of the project. The two criteria are explained in 
the following paragraph. 

A. PROJECT RANKING CRITERIA 

The two major considerations when deciding upon which 
projects to expend research resources are the potential 
Military Significance of the program and the Investment 
Risk of the program. The judging groups of this study 
were asked to construct priority ladders of the con- 
sidered programs under each of those two considerations. 



9 



1 . 



Criterion A. Military Significance 



Military Significance refers to the needs of the 
Navy. The projects were ranked on the basis of which 
would yield the greatest payoff to the Navy if the re- 
search was successful. Those projects thought to have 
high military significance were ranked high and so on, 
down to those of low military significance. The proba- 
bility Qf a solution being achieved was not considered 
under this criterion, only the significance of the solu- 
tion if it were achieved. 

2 . Criterion B. Investment Risk 

Investment Risk refers to the probability of a 
successful research effort regardless of the value to 
the Navy of the payoff. The project orderings repre- 
sented each judge T s belief of which problems had a high 
probability of being successfully solved (ranked high) 
and which had a lower probability of being solved (ranked 
lower). A research effort was defined as successful if 
a solution to problems or significant improvements in 
the present state could be developed within a period of 
ten years. 

B. THE JUDGE GROUPS 

The judge groups chosen for this study represent 
those organizations which are vitally interested in the 
allocation of research resources to the various projects. 
The list of the nine groups is contained in Appendix A. 
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They represent a cross-section of persons directly con- 
nected with the Bureau of Naval Personnel and with the 
Naval Personnel Research Laboratories. Both managers and 
professional workers are included as judges. In addition, 
group number 7 is composed of persons who are users of 
research products; group number 8, evaluators of research 
efforts; and group 9, contracted workers on research pro- 
grams. Accordingly, none of these last three groups is 
directly involved in selecting projects for emphasis in 
r es ear ch . 

C. THE PROJECTS STUDIED 

The projects selected for study are ones of current 
high interest within the Department of the Navy. Appen- 
dix B lists and describes the projects. One of the 
objectives of this study was to determine if the judge 
groups felt that program priority lists should be re- 
assessed. The projects considered represent all those 
that Personnel Logistics (General Operating Requirement 
43) is currently concerned with, plus the additional 
project of R & D Resources Management (No, 16), 
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III. CALCULATIONS ON DATA 



The raw data submitted by each individual judge was 
in the form of two priority ladders, one for each rank- 
ing criterion. To facilitate comparing the judge groups, 
a single composite ranking of projects for each group, 
which represented the preferences of each judge in that 
group, had to be derived. 



A. THE FORD PROCEDURE 

The technique for developing a composite ranking for 
each judge group is known as the FORD PROCEDURE [Ref. 1]. 
Basically, the program determines a weighted value for 
each project. The weight statistic is calculated from 
a win-loss matrix by solving iteratively the equation 



n+1 
w . 



£ 



a . . 
i J 



£ 

i 



1J 



+ a 



JiL 



w , 



n 



+ w 



n 



where 

= the number of times project i was preferred to 
project j by all judges in the group. 

w*? = the weighted value assigned to project i on the n t h 
iteration . 

w^ = the win percentage for project i on the first itera- 
tion . 
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It should be noted that the FORD PROCEDURE produces 
a composite judgment that reflects the contribution of 
individual judges according to the number of judgments 
he makes . 

Several commendable characteristics which illustrate 
why the FORD PROCEDURE is especially applicable to this 
study are, 

1. A judge or ranker needed not rank all the pro- 
jects, only those he felt knowledgeable about and compe- 
tent to rank. 

2. Each judge could make his ranking as coarse or 
as fine as he desires. There was no restriction on the 
number of judgmental categories he may use. 

3. A judge had no requirement for a fixed distribu- 
tion of projects among the categories except that no more 
than one-third of the projects could be placed in any one 
category . 

4. A judge could indicate projects of tied ranking 
by placing them in the same category. 

The input to the FORD PROCEDURE, as run on the IBM/ 
360 computer, was composed of each individual judge*s 
ranking of some or all of the eighteen projects, ordered 
under Criterion A or B. The program considered all judge 
rankings of judges in one group and calculated a weighted 
value for each project. By ordering the projects with 
the highest weight first, a single composite ranking of 
projects representing all the judges in that group was 
obtained . 
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B. DATA MATRIX TRANSFORMATION 



After computing a composite ranking for each group, 
the data for analysis had been reduced to an 18 x 18 
matrix M . ’Rank positions’ were labels on the ordinate 
and ’judge groups’ were the labels on the abscissa. One 
additional ranking was added to the data matrix as column 
10. It represented the projects ranked high to low as 
reflected by current fiscal year funding. Columns 1 
through 9 of matrix M represented the priority rankings 
of the projects by the judge groups under the A criterion. 
Columns 11 through 19 represented the priority rankings of 
the judge groups 1 through 9 respectively, under criterion 
B. Element m_ showed which project was ranked in the 
i tji position by judge group j. Appendix C contains 
matrix M. 

To enable easier calculation of correlation coeffi- 
cients between judge groups and between projects, the 
matrix M was transformed to a matrix N, 18 x 19. Matrix 
M had ’project number’ on the ordinate and ’judge group’ 
on the absicssa. Thus the element n_ represented the 
ranking of project i by judge group j. 
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C, CORRELATION BETWEEN JUDGE GROUPS 

From Matrix N the correlation between judge groups 
was calculated using the formula 



where 



r . . 



1 



N(N 2 -1) 



2 



^ij = difference between the ranking of project k by 

judge groups i and j. 

N = the number of ranks, 

k = an index to facilitate stepping through each of the 
18 projects. The correlation coefficient r_ has the ad- 
vantage of not requiring any assumption on the distribu- 
tion of the data. Matrix R in Appendix D displays the 

r . . values . 
iJ 



D. PROJECT COLLATION 

A statistic was required that would quantify the rela- 
tionship of project i to project j. It was decided that 
the best method was to compare their respective rankings 
by the judge groups. By calculating a difference in the 
mean ranking of each project, no consideration would be 
given to the variance of the rankings. Note that if pro- 
jects r and s with mean rank difference equal to X have 
small rank variance, they are ’farther apart’ than projects 
u and v with a difference in mean ranks equal to X and with 
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large rank variance. Therefore, the statistic 
developed and used as a dissimilarity measure. 




was 




MR. - MR. 

L _-JLl 

/ (STD ± ) (STD) ~ 



where 



MR_^ = the mean rank of project i over the judge group 
rankings . 

STD_^ = the standard deviation of the rankings of project 
i. Project mean ranks, standard deviation of the rank- 
ings, and the tabled t_ statistic are shown in Appendix 
E. The smallest value for standard deviation of project 
ranking was 1.68; therefore, the problem of a very small 
STD_^ value disproportionately affecting the t __ statis- 
tic was not encountered. 
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IV. MULTIDIMENSIONAL SCALING ANALYSIS 



The method chosen to compare the various judge groups 
and to compare the projects was multidimensional scaling 
analysis. The particular technique employed was non- 
metric Smallest Space Analysis (SSA), and has evolved 
through studies by Shepard, Kruskal, Guttman and Lingoes 
[Refs. 2-6], The computer program used was developed at 
the University of Michigan by Guttman and Lingoes and is 
designated Smallest Space Analysis - I (SSA-I) [Ref. 7]. 
The program finds a set of coordinates for each of the n 
points (n judge groups) in an ra dimensional Euclidean 
space such that the distance between points is a mono- 
tonic function of inter-judge group correlation. More 
precisely, given a set of n objects for which there is a 
defined distance function, such as object correlation 
ranks on the n(n-l)/2 pairs of points, find that set 

of real numbers, x. (i = 1, 2,.., n; a = 1, 2,.., m) 

x a 

such that if 



distant function p was important to the scaling analysis 
and deserves amplification. When reading as computer 



Lingoes, J. C., New Computer Developments in Pattern 
Analysis and Nonmetric Techniques, paper presented at 
the 1964 IBM Symposium of Statistics, Paris: Gauthier- 
Villas, 24 November 1966. 




m 



k=l 



then d 



< d^ whenever p_^_. < • The concept of the 
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input the correlation matrix R, the distance function p_ 

refers to the rank of correlation coefficient r.. in an 

iJJ 

ordered column vector V, For example, the highest r^_. 

value was assigned - 1, the second highest was 

assigned p^ = 2 etc. The meaning of f d_ < whenever 

p. . < p, , ’ is then obvious, 

ij kl 

The practice of utilizing only the ranks of the co- 
efficients and not the actual values when determining 
optimal point orientation on the multidimensional plot is 
the nonmetric method. Nonmetric rather than metric con- 
straints permit bypassing strong linear and distributional 
assumptions required of metric methods. Defining a true 
metric on psychological data, as was the case in this 
problem, was impossible. To enable the plotting of a 
meaningful display the number of Euclidean dimensions is 
limited t-o m = 2. A true metric constraint denies that 
possibility due to the strict linear distance measure- 
ments between points, A nonmetric toonotonic distance 
function allows for the reduction to a two dimensional 
display while still maintaining a meaningful plot. 

Several goodness-of -f i t coefficients used in this study 
confirm the nonmetric assumption as acceptable. 



A. SMALLEST SPACE ANALYSIS (SSA-I) ALGORITHM 

The algorithm for determing the optimal orientation 
of points representing the n objects proceeds as follows. 
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STEP 1: Read in the data matrix R of inter-group 

correlation values or the data matrix T of inter-project 
dissimilarity values. 

STEP 2: Sort the data values from large to small if 

similarities and small to large if dissimilarities. 

Assign ordered ranks of 1 to n(n-l)/2 to the ordered 

vector. Indicate the rank of element r.. by p... 

ij iJ 

STEP 3: Construct a matrix C by 



c 



ij 




P ± j / [n (n-1) / 2 ] 



£ c u 

j=i 



j 

i = j 



Solve for the Eigenvectors and Eigenvalues of C. The 

set of Eigenvectors x. ‘(i = 1, 2,...,n; a = 1, 

l a 

is the initial orientation of the n points in the m 
dimensional Euclidean space. 

STEP 4: Calculate the inter-point distances over 

the m dimensional space. 



m 



^ - v 2 ’ 1 ' 2 



Order the distances d_ in a row vector with n(n-l)/2 
element s . 



STEP 5: Redefine the p f s as the d f s permuted to 

maintain the rank order of the original distance function 
and designate them d*. 
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* 



STEP 6: Calculate a measure of effectiveness, the 

phi coefficient. 



<p 



n-1 

z 

i = l 



n 



E 

j=i+i 



(d 



ij 



- d. .) 
ij 



STEP 7: Calculate a new C matrix on the basis of 

the d 1 s and the d* f s. 



c . . = — 



n 



1 + s 

k=l 



1 - ^ 
ij 









* ii 

where d._. = 0, d.. = 0, and — = 0 by convection. 



n 



1 1 



ii 



STEP 8: Calculate a new set of point coordinates 

n 



x^ 1 = 1/n E c . . 

lk j=1 ik ij 



1c 1,2, . . . , m 



STEP 9: Calculate again the inter-point distances 

as per step 4 and repeat steps 5, 6, 7 and 8. Continue 

until the normalized phi coefficient indicates further 
adjustment of points will not advance the state of the 
inter-point distance and original distance function 
relationship toward actual mono t onici ty . 



B. GOODNESS-OF-FIT MEASUREMENTS 

1 . Gut tman-Lingoes Coefficient of Alienation 

Consider a scattergram of n(n-l)/2 points 
& 

coordinates vs d„ . Maximizing a statistic 



on 

y 
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is defined as minimizing the alienation of plotted points 

from a regression line through the origin. The coeffi- 

1/2 

cient of alienation is K = (1 - y) . The statistic y 

* 

is a rank-order correlation coefficient between p. . = d, . 

iJ ij 

* 

and d. . • y equals 1 when d. . = a d for some constant 

iJ iJ iJ 

of proportionality a and it follows that the coefficient 
of alienation K would equal zero. 

2 . Kruskal 1 s Stress 

Kruskal’ s stress is derived from a t wo -d imens ion- 
* 



al plot of d_^ vs and is a statistic quantifying how 

well the given configuration of points represents the 
data. A monotonic increasing piece-wise linear line is 
drawn connecting the points. The .points d. . are also 
plotted on the graph. The distance ( d _ - d _ ) is defined 

as the horizontal difference between the actual distance 



A 

d. . and the desired distance d, . . Kruskal s stress is 

ij ij 

defined as, 




Application of the statistic has shown that for any value 
of S less than .20, the plotted configuration of points 
is a good representation of the original data. Zero 
stress indicates complete compliance with the monotonic 
cons t r aint . 
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V. 



CALCULATION RESULTS 



To accomplish the objectives of this study of compar- 
ing the judge groups under each of the ranking criteria, 
and of comparing the projects being ranked, the Smallest 
Space Analysis - I was employed several times. 

A. JUDGE GROUP COMPARISON 

1 . Comparison of Judge Groups under Criterion A 

Columns 1 through 10 of matrix N in which n__ rep- 
resented the ranking of project i by judge group j under 
Criterion A (Military Significance) and column 10 repre- 
senting fiscal year funding, were taken for the first 
calculation. The correlation matrix R was calculated for 
the measure of similarity between judge groups. r_(i=l, 
2,. ..,10; j=l, 2,... ,10) was then used as input to SSA-I. 

Table I shows the resulting point positions rep- 
resenting the optimal orientation of each judge group 
with respect to the other nine. The values for Guttman- 
Lingoes’ coefficient of alienation of .290 and Kruskal's 
stress of .243 were both marginal values and indicated 
that an actual monotonic relationship between similari- 
ties and inter-point distances was not attainable. The 
plot of points, Figure 1, did not indicate any clustering 
or significant similarity among judge group rankings. 
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TABLE I 



PLOT OF JUDGE GROUPS 

RANKING UNDER THE MILITARY SIGNIFICANCE CRITERION 



Judge Group Plot Coordinates 



1 


6 .268 


59.664 


2 


-50-775 


-69 . 316 


3 


6.069 


-16 .553 


4 


-100 . 000 


-37.483 


5 


-80.282 


50.179 


6 


100 . 000 


-14 .333 


7 


19.227 


-100 . 000 


8 


36 . 415 


32 .546 


9 


-26.288 


20 .693 


10 


-59 . 613 


-7.008 


Gut tman -Lingoes ' 
Kruskal's Stress 


Coefficient of Alien 
= 0.24326 


ation = 
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Figure 1. Plot of Judge Groups Ranking under the 
Military Significance Criterion. 
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2 . Comparison of Judge Groups under Criterion B 



Columns 10 through 19 of matrix K with n__ repre- 
senting the priority rank of project i with respect to 
fiscal year funding and judge groups 1 through 9 respec- 
tively, under Criterion B were used for the second run. 

The correlation values r^_. (i=10 , 11 , . . . , 19 ; j =10 , 11 , . . . 19) 

from matrix R of Appendix D was the input to SSA-I, 

Table II shows the resulting point positions on 
the two-dimensional display. The values for Guttman- 
Lingoes ’ coefficient of alienation of .257 and for Krus- 
kal’s stress of .214 were again marginal. The plot of 
points in Figure 2, by being very scattered, indicated 
that none of the judge groups agreed to a significant 
degree on their project rankings, 

3 . Comparison of Judge Groups under Both Criteria 

Neither the plot of judge groups under Criterion 
A nor the plot of judge groups under Criterion B indica- 
ted a strong similarity in group decisions. By plotting 
all 19 "groups" together it would be possible to see if 
any groups submitted very similar rankings under both 
criteria. Correlation coefficients r.. ( i = l , 2 , . . . , 19 ; 

j =1 , 2 , . . . , 1 9) from matrix R of Appendix D were input to 
SSA-I . 

Table III lists the plot positions. The Guttman- 
Lingoesj coef f icient of alienation was .190 and the Krus- 
kal stress was .167. The values indicated an excellent 
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TABLE II 



PLOT OF JUDGE GROUPS 

RANKING UNDER THE INVESTMENT RISK CRITERION 



Judge Group 


Plot Coordinates 


i 


89.897 


38.945 


2 


33.497 


-94 . 012 


3 


-100 . 000 


-45.473 


4 


-83 .266 


57 . 879 


5 


33 . 039 


69 . 747 


6 


100.000 


-53.064 


7 


16.167 


-16 . 339 


8 


-23.906 


100.000 


9 


-1.639 


23.583 


10 


-17.311 - 


112 . 864 


Guttman-Lingoes ' 

Coefficient of Alienation = 0,25766 




Kruskal 1 s Stress - 


0.21421 
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Figure 2. Plot of Judge Groups Ranking under the 
Investment Risk Criterion. 
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TABLE III 



PLOT OF JUDGE GROUPS 
RANKING UNDER BOTH CRITERIA 



Judge Group Plot Coordinates 



1 


-18.158 


24 .269 


2 


-6 .882 


10.933 


3 


51.030 


-100 .000 


4 


-90.134 


-24.549 


5 


-21.517 


-80 .273 


6 


-8.888 


-40 .472 


7 


-64 . 799 


-59 . 218 


8 


-96.362 


-48 .063 


9 


-100.000 


-94 . 551 


10 


18.474 


-83.546 


11 


74.592 


-7.040 


12 


81.520 


-25 .029 


13 


86.038 


-42 . 891 


14 


59.298 


-25.100 


15 


66.162 


-65 .892 


16 


72.194 


-28.508 


17 


-9.576 


-16.639 


18 


62 .696 


-71.514 


19 


100.000 


-81.553 


Gut tman-Lingoes ' 
Coefficient of Alienation 


= 0.19045 





Kruskal's Stress = 0.16754 
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Figure 3. Plot of Judge Groups Ranking under Both 
Criteria . 
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plot in terms of variation from the regression line for 
the G-L coefficient as detailed in paragraph IV C 2. The 
Kruskal stress also indicated excellent correlation of 
plotted values with the monotonically increasing line on 
vs d^_. as explained in paragraph IV C 2 . 

Figure 3 was not applicable to clustering consid- 
eration because it represented the judge groups as nine- 
teen separate entities, i.e., each judge group was 
represented twice. However, one thing was clearly indi- 
cated. Each judge group submitted significantly differ- 
ent rankings for each of the two criteria. With the 
exception of group 3 under Criterion A (point 3) and 
group 7 under Criterion B (point 17), there was no over- 
lapping of the judge group rankings. The results indica- 
ted that the judges actually did consider the criteria 
of Military Significance and Investment Risk when ranking 
their projects. That is an important argument for data 
validity . 



B. PROJECT COMPARISONS 

The second objective of this study was to compare the 
projects and examine possible groups of projects that the 
judges considered of equal importance or that judge group 
consensus rated high or low. 

1 . Comparison of Projects Ranked under Criterion A 

The eighteen projects as ranked under Criterion A 
plus the fiscal year funds ranking were the first of the 
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project groups considered. The entries of matrix T in 

A 

Appendix E are values of the interpoint dissimilarity 
measure t^_. for the Military Significance Criterion. 

The input to SSA-I was the matrix T^. 

Table II shows the point position representing 
the projects on the tv/o-dimensional plot. The value of 
Gut tman-Lingoes 1 coefficient of alienation was .067 and 
Kruskal/s stress was .053, Both values indicate that the 
resulting plot was an excellent reflection of the origi- 
nal data as determined by the distance function p. . , 

ij 

Figure 4 showed definite clusters of projects. The rule 
chosen for defining a cluster of c points was that each 
point be within 20.0 units of the other c-1 points of the 
cluster. (The longest possible inter-point distance on 
the plot was 282.0 units.) Using that rule, 2 major 
clusters of 3 or more points were established. One 
cluster was composed of projects 5, 8 and 18, and the 

other cluster was composed of projects 3, 10, 12, 14 
and 15, Minor clusters were established between pro- 
jects 7 and 9, and between projects 4 and 16. The 
implication drawn from the clusters was that the pro- 
jects in each cluster were considered of equal priority 
by the judge groups when ranking under the Military 
Significance Criterion. The SSA-I plot is more meaning- 
ful than a standard correlation statistic because each 
point was plotted with respect to the other n-1 points. 
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TABLE IV 



PLOT OF PROJECTS 

WHEN RANKED UNDER THE MILITARY SIGNIFICANCE CRITERION 



Project No. Plot Coordinates 



1 


88.636 


-46 . 802 


2 


-80.882 


22.898 


3 


5 . 479 


-82 .690 


4 


50.223 


-48.019 


5 


-34.760 


-52 .799 


6 


-16.022 


-75.868 


7 


38.913 


-67.702 


8 


-41.815 


-56 . 905 


9 


30.572 


-66.946 


10 


16.792 


-66 . 719 


11 


-81.456 


-100 .000 


12 


3.653 


-77.032 


13 


100.000 


-26.187 


14 


11.188 


-74.394 


15 


16.804 


-87.316 


16 


55.806 


-37 .771 


17 


-100.000 


-51.790 


18 


-51.634 


-49.414 


Gut tman-Lingoes ' 
of Alienation 


Co ef f i cient 
= 0.06728 




Kruskal's Stress 


= 0.05308 
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Figure 4. Plot of Projects when Ranked under the 
Military Significance Criterion, 
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A correlation statistic reflects the collation between 
two points regardless of each point's relationship to 
the remaining points. 

2 • Comparison of Pro jects under Criterion B 

The entries of matrix Tg in Appendix E were used 
as the SSA-I input for comparing projects as they were 
ranked by the judges under the Investment Risk Criterion. 

Table V shows the point positions, the Guttman- 

I 

Lingo e s 1 coefficient of alienation as *030 and the Kruskal 
stress as ,021, Again the values indicated an excellent 
fit with respect to the plotting constraints. Figure 5 
showed the project point orientation. Using the rule 
that inter-point distances be less than 20,0 units for 
each cluster point, three major clusters were established. 
One cluster was composed of projects 3, 15, 17 and 18; 

another was composed of projects 1, 13 and 14, and the 

third of projects 6, 7 and 10. A minor cluster between 

projects 12 and 16 was also in evidence. Therefore, rank- 
ing on the basis of Investment Risk the judges felt the 
clustered projects entailed similar risk. 

3 . Comparison of Projects using Both Crijterija 

In actual practice both Military Significance and 
Investment Risk are considered when allocating research 
resources. For that reason a plot of the projects consid- 
ering both criteria was* desired. Matrix of Appendix E 

was used as the input for SSA-I. It was derived by con- 
sidering the judges as nineteen separate groups*. 
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Table VI displays the point positions. The Gutt- 
man-Lingo es ? coefficient of alienation of .026 and the 
Kruska'l stress of .018 both indicated an excellent plot 
in terms of meeting the plotting algorithm constraint 
criteria. Figure 6 showed several clusters, the major 
ones being projects 5, 8, 17 and 18, and projects 4, 6 

and 7. Minor clusters consisted of points 9 and 11, points 
3 and 15, points 12 and 14, and points 1 and 16. Consider- 
ing the projects under both ranking criteria, those 
clusters represented the projects which the judge groups 
felt should receive equal emphasis for research. 
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TABLE V 



PLOT OF PROJECTS 

WHEN RANKED UNDER THE INVESTMENT RISK CRITERION 



Project No, 


Plot Coordinates 


i 


81.826 


-31 . 333 


2 


-85.361 


15 . 915 


3 


- 54.677 


-66 . 792 


4 


-68.344 


-52.507 


5 


-100.000 


-29 . 425 


6 


-18.818 


-74 . 158 


7 


-30.314 


-64.395 


8 


-67 . 295 


-76 . 211 


9 


-82.586 


-4.289 


10 


-20.116 


-87 .728 


11 


26.285 


-100 .000 


12 


71 . 931 


-43.641 


13 


100 . 000 


-23.824 


14 


94.434 


-27.973 


15 


-48.304 


-61.729 


16 


68 . 200 


-54.758 


17 


-45.503 


-70 . 721 


18 


-43.436 


-64 . 363 


Guttman-Lingoes 
Coe f f i cien t of 


t 

Alienation = 0 , 


03073 



Kruskal's Stress = 0.02190 
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Figure 5. Plot of Projects when Ranked under the 
Investment Risk Criterion. 
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TABLE VI 



PLOT 
WHEN RANKED 

Project No . 

1 

2 

3 

4 

5 

6 
_ 7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 



OF PROJECTS 

UNDER BOTH CRITERIA 



Plot Coordinates 



84 . 667 


-38.574 


-100.000 


46 . 714 


-55.133 


-88.801 


-30.757 


-83 . 865 


-89.757 


-35 . 684 


-37.145 


-96.417 


-21.369 


-88.825 


-96.979 


-66 . 706 


-58.968 


-63 . 651 


-13.578 


-100 . 000 


-66 . 832 


-64 . 756 


29.745 


-75.383 


100.000 


-9 . 750 


34.864 


-67.964 


-50.375 


-90.462 


77.295 


-52 . 623 


-96 . 708 


-47 . 245 


-85 . 638 


-59.449 



ttman-Lingoes 1 

oefficient of Alienation = 0.02633 
Kruskal's Stress = 0.01814 
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Figure 6. Plot of Projects when Ranked under Both 
Criteria . 
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VI . CONCLUSIONS 

Results of the SSA-I plots of judge groups indicated 
that no two judge groups were in strong agreement when 
considering their rankings as a whole. Groups often 
agreed on which projects should be ranked high under one 
criterion or the other but disagreed on which projects 

t . > 

should be ranked low. The reverse, agreeing on the lower 
ranked projects but disagreeing on the higher ranks, was 
also in evidence. 

Different conclusions can be drawn from the project 
comparisons. The three plots using Criterion A, Crite- 
rion B, and both Criteria A and B, all displayed cluster- 
ing tendencies. To display the results in an easy-to-read 
fashion the following was done. For each cluster, the 
mean rank of the projects in that cluster was calculated 
and designated the mean cluster rank. A ladder was con- 
structed of ordered mean ranks for both clusters and non- 
clustered projects. The result is how the judge groups 
felt the projects should be ranked considering the three 
criteria above. Figure 7 displays the resulting project 
priority ladders. 

Investigation of the variation in project rankings 
(Appendix E) shows which project was ranked most consis- 
tent by the judge groups. That project with the lowest 
standard deviation of ranking was project 17 with a mean 
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rank of 4.70 and a standard deviation of 1.68 for Crite- 



rion A, project 13 with a mean rank of 15.90 and a 
standard deviation of 1.87 for Criterion B, and project 
13 with a mean rank of 15.74 and a standard deviation 
of 2.27 for Criteria A and B together. 
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Rank 



Project Title 



1 

2 

3 

4 



5 

6 



7 

8 

9 



P r o j . No. 

2 

11 

17 

5 

8 

18 

6 

3 

10 

12 

14 

15 

7 

9 

4* 

16 



1 



Education and Training 

Personnel Retention 

Recruitment/Procurement 

Manpower Requirements 
Selection/Classification 
Personnel Performance 

Personnel Management Systems 

Occupational Research and 
Development 
Career Structures 
Attitude/ Motivation 
Organizational Effectiveness 
Distribut ion /Assignment 

Man-Machine System R 6 D 
Applied Systems Development 

Manpower /Personnel Cost Systems 
R & D Resources Management 

Drug Abuse 



10 



13 • 



Minority Discrimination 
Measurement and Reduction 



Figure 7a. Project Priority Ladder as a Result of 
Clustering under the Military Significance Criterion. 
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Rank 



P r o j . No. 



Project Title 



1 



2 



Education and Training 



2 



9 



Applied Systems Development 



3 



5 



Manpower Requirements 



4 



4 



Manpower/Personnel Cost Systems 



5 



8 



Selection/ Classification 



6 



7 



3 

15 

17 

18 

f 

6 

7 

10 



Occupational Research and 
Development 

Distribution/ Assignment 
Recruitment /Procurement 
Personnel Performance 

Personnel Management Systems 
Man-Machine Systems R & D 
Career Structures 



8 



11 



Personnel Retention 



9 



12 

16 



Attitude/Motivation 
R & D Resources Management 



10 



1 

13 

14 



Drug Ab use 

Minority Discrimination 

Measurement and Reduction 
Organizational Effectiveness 



Figure 7b. Project Priority Ladder as a Result of 
Clustering under the Investment Risk Criterion. 
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Rank P r o j . No. 



Pr o j ect Title 



1 



2 



Education and Training 



8 

17 

18 



Manpower Requirements 
Selection/ Classification 
Recruitment /Procurement 
Personnel Perf ormance 



3 




Applied Systems Development 
Personnel Retention 



4 



5 




Occupational Research and 
Developmen t 

Distribution/Assignment 

Manpower/Personnel Cost Systems 
Personnel Management Systems 
Man-Machine Systems R & D 



6 10 



Career Structures 



7 




Attitude/Mot ivat ion 
Organizational Effectiveness 



8 




Drug Abuse 

R & D Resources Management 



9 13 



Minority Discrimination 

Measurement and Reduction 



Figure 7c, Project Priority Ladder as a Result of 
Clustering under both Criteria; Military Significance 
and Investment Risk, 
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APPENDIX A 



LIST OF JUDGE GROUPS 



Group Composition 

Manager personnel, Pers A-3 

Manager personnel, Headquarters, Dept, 
of the Navy 

Manager personnel, Naval Personnel 
Laboratory, Washington, D. C. 

Professional worker, Naval Personnel 
Laboratory, San Diego 

Professional worker, Naval Personnel 
Laboratory, Washington, D. C. 

Manager personnel, Naval Personnel 
Laboratory, San Diego 

Members, user panel (students at the 
Naval Postgraduate School) 

Members, Laboratory Advisory Board 
for Personnel Laboratories 

Technical /Professional persons, con- 
tractor activities 

Fiscal year 1973 dollar funding for 
the specified project 



APPENDIX B 



LIST OF PROJECTS STUDIED 



This Appendix lists the projects studied by identi- 
fication number and short and includes a paragraph 

describing each. This information was available to each 
judge when he compiled his ranking. 



Project No. Short Title 

1 Drug Abuse 

This area consists of research and develop- 
ment for (a) determining causes, attitudes toward, and 
extent of drug abuse in the Navy; (b) exploration of means 
to reduce, prevent, and control drug abuse; and (c) means 
to provide effective education, counseling, and rehabili- 
tation programs for Naval personnel and dependents. 

2 Education and Training 

The two mutually supporting purposes of 
Education and Training research and development are (a) 
improving Navy training and (b) development of principles 
and technology of training. Navy requirements call for 
more accurate definition of training requirements and 
for training methods directed toward meeting future job 
requirements. Research and development in this area in- 
volves work on the broad spectrum of problems related 
deriving and specifying appropriate training content and 
developing and evaluating ways to improve acquisition 
of training objectives. Research in this area also 
includes specific problems of learning, perception, 
job aids, programming, displays, simulation, and manage- 
ment as they relate to the improvement of training. 

3 Occupational Research and Development 

Development, test and evaluation of new 
and improved techniques for describing, evaluating, 
structuring, and re-engineering Navy occupations for 
optimal matching of available manpower characteristics 
to changing job requirements. Involves new methods for 
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determining skills required for effective performance of 
occupations and the storage and retrieval of occupational 
data pertaining to work requirements of Navy personnel. 

4* Manpower /Personnel Cost Systems 

Development, test and evaluation of new 
and improved (a) techniques for the acquisition, computa- 
tion, storage, and retrieval of manpower /per sonnel cost 
data; (b) techniques for determining the cost of various 
mixes of men and equipment, personnel management policies, 
and personnel utilization factors; and (c) .measures which 
permit manpower requirements planners to trade off cost 
and effectiveness in order to evaluate available options. 

The principal objective is to furnish total government 
costs from procurement through p o s t -s epar at i on which can 
be used to formulate cost-related decisions concerning the 
total Navy manpower personnel systems, 

5 Manpower Requirements 

Development, test and evaluation of 
techniques for defining and displaying quantitative and 
qualitative manpower requirements for all functions per- 
formed ashore and afloat including upward aggregation of 
requirements to the total Navy establishment. The tech- 
niques must be responsive to the changing operational 
requirements, system design innovations, and available 
human resources. Includes the development of manpower/ 
productivity models and the use of simulation techniques 
to test alternative manning and manpower management con- 
cepts in terms of cost and total system effectiveness. 

The principal objective is to develop methods for deter- 
mining manpower requirements which are capable of respond- 
ing quickly to changes in resources and military objectives 
and requirements. 

6 Personnel Management Systems 

Development, test and evaluation of tools 
to improve the quality of personnel management by increas- 
ing the accuracy of force projections, by responding more 
rapidly to changing policies and operational conditions, 
and by projecting the effects of current and proposed per- 
sonnel policies to determine the most effective personnel 
management strategies. These tools are usually in the form 
of computerized models of the personnel system which can be 
used to reflect the relationship between such variables as 
recruit input, advancement, and attrition and to project the 
effect of these variables upon the size and skill composi- 
tion of the personnel force. 
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7 



Han-Machine R & D 



This area is primarily concerned with the 
implementation of human operator considerations in the 
development, operations, and maintenance of new and cur- 
rent organizational, weapons and support system's. Goals 
include (a) an understanding of, and techniques to quantify 
and extrapolate, meaningful relationships among human per- 
formance, equipment characteristics, and environmental 
conditions; (b) development and application of man-machine 
standards, specifications, and tradeoff analysis methodolo- 
gies in systems development, planning, and procurement; and 
(c) techniques for application of human capabilities to 
anticipated demand of future weapon systems. 

8 Selection/Classification 

The program area involves development, 
test and evaluation of new and improved methods for (a) 
more accurate and effective selection and classification 
of individuals; (b) determining their training requirements; 
and (c) matching the man to the job. Selection usually 
refers to the process of determining which individual or 
individuals should be accepted or rejected with regard to 
a specific program. Classification refers to determining 
and categorizing the appropriate skill qualifications of 
an individual that each school or job quota is filled 
with particular individuals most likely to perform success- 
fully . 

9 Applied Systems Development 

Identification of personnel and training 
requirements, establishment of initial operator performance 
standards and maintenance skill requirements, assistance 
in man-machine tradeoff studies, and conducting studies to 
enhance man-machine compatibility for candidate and proto- 
type systems under development. The effort frequently 
involves development of training objectives, review of 
course and lesson plans, and the evaluation of operator 
and maintenance personnel training. 

10 Career Structure 

Development, test and evaluation of new 
and improved techniques for (a) arriving at optimal career 
progression patterns; (b) methods for utilizing occupational, 
manpower /personnel cost, and other manpower factors data 
in determining optimal combination of grade and skill 
profiles required to satisfy Navy operational requirements; 
and (c) techniques for providing balance between needs of 
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the service based on technological projections, and needs 
of the individual related to opportunities for increased 
responsibility, advancement, prestige, and compensation. 
Typical outputs are optimum length of careers, optimal 
level of specialization, and optimum amount of fleet ex- 
perience required of each grade of occupational specialty 
to meet changing military requirements. 

11 Personnel Retention 

This is a multi-faceted program exploring 
and developing methods and techniques for inducing quali- 
fied and needed personnel to remain on active duty, and 
programs which meet fluctuating needs of the naval service 
Included are (a) investigation in psychometric technology 
to predict retention for use in selection and assignment; 
(b) development of more fair and effective evaluation of 
performance through computer-aided analysis and display of 
performance evaluation; and (c) development of improved 
techniques for selection and training of personnel in 
ratings which administer personal services to other naval 
p er s onn e 1 . 

12 Attitude /Motivation 

Develop sociometric and psychometric in- 
struments and conduct controlled experiments for purposes 
of studying man’s values, needs, aspirations, prejudices 
and expectations to achieve a better understanding of how 
they affect his thinking, feelings, and reactions to 
people, groups, social issues or events in his environment 
The objective in achieving such an understanding is better 
prediction and control of an individual’s behavior, moti- 
vation and performance. 

13 Minority Discrimination Measurement and 

Reduction 

This area is comprised of R & D support 
for BuPers and Navy objectives to intensify and expand 
efforts to eliminate racial, sex, and cultural discrimina- 
tion and bias in the Navy. 

14 Organization Effectiveness 

Development and evaluation of techniques 
for measuring overall military effectiveness of organiza- 
tional units under various contingencies. This involves 
the development of empirically based criteria and the 
relative importance of criteria which should be applied 
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in making judgments of organizational effectiveness. It 
includes studies of individual and organizational behavior 
and evaluative research into managerial practices and or- 
ganizational characteristics. The principal objective is 
the enhancement of organizational effectiveness in rela- 
tion to organizational goals and total system effectiveness. 

15 Distribut ion /Assignment 



development in the managerial functions of personnel 
distribution, assignment, and rotation as carried out by 
the Bureau of Naval Personnel. The program is concerned 
with the development and evaluation of new and improved 
techniques and procedures based on modern computer 
technology for (a) the equitable, quantitative and 
qualitative allocation ("distribution”) of the personnel 
and inventory among the ships, aircraft squadrons, and 
shore stations of the Navy; (b) the optimal match 
("assignment") of specific individuals to particular 
billets in terms of prescribed criteria; and (c) the 
planned, periodic reassignment ("rotation") of career 
personnel to the various categories of sea, shore and 
overseas duty. Objectives are the optimum matching 
of personnel and billets, improved manpower utilization 
systems, and optimal matching of manpower requirements 
and resources over time. 

16 R & D Resources Management 

This area includes three subareas dealing 
with efficient use o f R & D resources. One deals with 



sonnel employed by the laboratories. Research in this 
area is aimed at the development of performance evaluation 
criteria in the area of quantitative and qualitative 
work output and scientif ic/ technical contributions. 

Another subarea concerns the identification of significant 
problem areas, assignment of priorities, and selection of 
tasks to be performed to assure maximum return for the"R & 
D dollar". A third subarea is the continuing study of the 
implementation of research recommendations by consumers as 
a basis for future research planning and to facilitate the 
implementation of significant R & D products. 

17 Recruitment/Procurement 

Development, test and evaluation of 
methods and techniques for (a) attracting and obtaining 



This area encompasses research and 



the selection and evaluation of 




pet- 
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required skill composition, racial mix, and educational 
levels and for (b) improving selection policies and 
programs relating to manpower requirements, implications 
of an all volunteer force, manpower ceilings reductions, 
and changes in strategic guidance. The principal 
objective is to obtain personnel with the appropriate 
innate and acquired characteristics to meet all of the 
performance requirements and changing requirements of 
future naval forces. 

18 Personnel Performance 

Development of on-the-job performance 
standards and criteria and improved proficiency measure- 
ment techniques to assure that selection, assignment, 
and promotion factors are job-relevant , The goal is 
maximum readiness of individuals and units. 
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APPENDIX C 



MATRIX M 

that project ranked in the I ^ position by judge group j 
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APPENDIX D 



JUDGE GROUP CORRELATION MATRIX R 
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APPENDIX E 



DATA ON INDIVIDUAL PROJECT RANKINGS FOR SPECIFIED CRITERION 



Criterion A Criterion B Criteria A&B 
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